{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question Answering Demo Application"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import html\n",
    "import pandas\n",
    "import pickle\n",
    "import json\n",
    "import spacy\n",
    "import warnings\n",
    "from IPython.display import display,HTML\n",
    "warnings.filterwarnings('ignore') #Some operations warn inside a loop, we\"ll only need to see the first warning\n",
    "sys.path.append(\"../..\")\n",
    "from aips import *\n",
    "engine = get_engine()\n",
    "outdoors_collection = engine.get_collection(\"outdoors\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "nlp.remove_pipe(\"ner\")\n",
    "def get_query_from_question(question):\n",
    "    words = [token.text for token in nlp(question)\n",
    "             if not (token.lex.is_stop or token.lex.is_punct)]\n",
    "    return \" \".join(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Listing 14.15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def display_contexts(contexts):\n",
    "    display(HTML(pandas.DataFrame(contexts[0:10]).to_html(index=False)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def retriever(question):\n",
    "    contexts = {\"id\": [], \"question\": [], \"context\": [], \"url\": []}\n",
    "    query = get_query_from_question(question)\n",
    "    request = {\"query\": query,\n",
    "               \"query_fields\": [\"body\"],\n",
    "               \"return_fields\": [\"id\", \"url\", \"body\"],\n",
    "               \"filters\": [(\"post_type\", \"answer\")],               \n",
    "               \"limit\": 5}\n",
    "    docs = outdoors_collection.search(**request)[\"docs\"]\n",
    "    for doc in docs:\n",
    "        contexts[\"id\"].append(doc[\"id\"])\n",
    "        contexts[\"url\"].append(doc[\"url\"])\n",
    "        contexts[\"question\"].append(question)\n",
    "        contexts[\"context\"].append(doc[\"body\"])\n",
    "    return pandas.DataFrame(contexts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>id</th>\n",
       "      <th>question</th>\n",
       "      <th>context</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>18376</td>\n",
       "      <td>What are minimalist shoes?</td>\n",
       "      <td>Minimalist shoes or \"barefoot\" shoes are shoes that provide your feet with some form of protection, but get you as close to a barefoot experience as possible. The styles range from simple ultralight sneakers, to thin almost slipper like shoes, all the way down to stick on shoes: image source image source</td>\n",
       "      <td>https://outdoors.stackexchange.com/questions/18376</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18370</td>\n",
       "      <td>What are minimalist shoes?</td>\n",
       "      <td>There was actually a project done on the definition of what a minimalist shoe is and the result was \"Footwear providing minimal interference with the natural movement of the foot due to its high flexibility, low heel to toe drop, weight and stack height, and the absence of motion control and stability devices\". If you are looking for a simpler definition, this is what Wikipedia says, Minimalist shoes are shoes intended to closely approximate barefoot running conditions. 1 They have reduced cushioning, thin soles, and are of lighter weight than other running shoes, allowing for more sensory contact for the foot on the ground while simultaneously providing the feet with some protection from ground hazards and conditions (such as pebbles and dirt). One example of minimalistic shoes would be the Vibram FiveFingers shoes which look like this. Image Source</td>\n",
       "      <td>https://outdoors.stackexchange.com/questions/18370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16427</td>\n",
       "      <td>What are minimalist shoes?</td>\n",
       "      <td>One summer job, I needed shoes to walk on a rocky beach, sometimes in the water, for 5 to 10 miles per day all summer. Stretchy neoprene shoes were terrible for this- no traction and no support. So I used regular sneakers. I chose a pair of cross country racing flats. The model I used are no longer made, but were made by New Balance. The uppers were extremely well ventilated polyester, so they drained very quickly, and the thin material dried much faster than a padded sandal, and certainly much faster than a regular sneaker of leather or cotton would. There are probably other minimalist sneakers that would be equally good. The thing to look for is thin fabric that attaches directly to the sole, with no rubber rim that would keep the water from draining. Running shoes in general will be good because those tend to be very well ventilated, and minimalist shoes will be best because there will be very little padding to hold extra water.</td>\n",
       "      <td>https://outdoors.stackexchange.com/questions/16427</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18375</td>\n",
       "      <td>What are minimalist shoes?</td>\n",
       "      <td>The answer to this question will vary on your level of foot strength and the terrain. Human beings did pretty well for thousands of years without supportive footwear, but when they started walking on hard stone and climbing rocky mountains they quickly discovered their feet weren't suited for it. You'll notice that most large mammals that live at high elevation have hooves. Most people entering the minimalist barefoot shoe niche grew up wearing sneakers that provided some level of support for your arch/ankle/etc., which has the byproduct of under-developing the stabilizer muscles in your feet as you grow and mature. The result is very sore feet after using the minimalist shoes for long periods of time; the risk is causing a muskuloskeletal disorder from straining these muscles by using them excessively before they have the opportunity to properly develop. It's not unhealthy to wear minimalist footwear, but on what terrain your wear them could be bad for your body in the long run. Human beings were never meant to walk or run exclusively on hard pavement or sidewalks. Nor were we designed to clamber around on sharp rocks at high elevations. Homo Sapiens are a low-elevation tropical animals, You'll notice the vast majority of the world's population is still in Southern Asia, which is very tropical. We had to adapt technology like clothes and footwear in order to thrive in other environments. If you're running on soft ground and you have the foot strength, then there are plenty of arguments in favour of minimalist shoes being better for you than other shoes, because it brings your posture and gait back to what nature intended it to be. If you're hiking in the mountains on uneven rocky terrain, especially while carrying a heavy bag, then you'd be better off wearing a supportive hiking boot, emulating a sturdy hoof of a high-altitude ungulate.</td>\n",
       "      <td>https://outdoors.stackexchange.com/questions/18375</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13540</td>\n",
       "      <td>What are minimalist shoes?</td>\n",
       "      <td>Barefoot Shoes Also known as minimalist shoes, the theory behind them is to give you as close to a barefoot feel without actually putting the soft, soft flesh of your terribly urbanized and unadapted soles of your feet at risk of damage. My favourite barefoot shoes are Vibram Five-Fingers , I wear either my Mouri's or my KSO's at the beach. Vibram Five-Fingers 'Signa' watersport shoes: The thin sole will be enough to protect your feet from the lava sand, and it will still feel like running on the beach barefoot, with the exception of your feet getting exfoliated by the sand. They also have running specific five fingers if you're interested.</td>\n",
       "      <td>https://outdoors.stackexchange.com/questions/13540</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "example_contexts = retriever('What are minimalist shoes?')\n",
    "display_contexts(example_contexts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Listing 14.16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "def get_processor_device():\n",
    "    return 0 if torch.cuda.is_available() else -1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import pipeline\n",
    "\n",
    "device = get_processor_device()\n",
    "model_name = \"data/question-answering/roberta-base-squad2-fine-tuned\"\n",
    "qa_nlp = pipeline(\"question-answering\", model=model_name,\n",
    "                  tokenizer=model_name, device=device)\n",
    "\n",
    "def reader(contexts):\n",
    "    answers = []\n",
    "    for _, row in contexts.iterrows():\n",
    "        answer = qa_nlp({\"question\": row[\"question\"],\n",
    "                         \"context\": row[\"context\"]})\n",
    "        answer[\"id\"] = row[\"id\"]\n",
    "        answer[\"url\"] = row[\"url\"]\n",
    "        answers.append(answer)\n",
    "    return answers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Listing 14.17"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def reranker(answers):\n",
    "    return sorted(answers, key=lambda k: k[\"score\"], reverse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Listing 14.18"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib.parse\n",
    "def print_answer(question, reranked):\n",
    "    url = \"https://outdoors.stackexchange.com/search?q=\" + urllib.parse.quote(question)\n",
    "    html = f'<h2><a href=\"{url}\" target=_blank>{question}</a></h2>'\n",
    "    for answer in reranked:\n",
    "         html += f'<h3><a href=\"{answer[\"url\"]}\" target=_blank> {answer[\"id\"]} </a>'\n",
    "         html += '<strong>' + answer[\"answer\"] + '</strong><em> (' + str(round(answer[\"score\"], 3)) + ')</em></h3>'\n",
    "    display(HTML(html))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ask(question):\n",
    "    documents = retriever(question)\n",
    "    answers = reader(documents)\n",
    "    reranked = reranker(answers)\n",
    "    print_answer(question, reranked)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h2><a href=\"https://outdoors.stackexchange.com/search?q=What%20is%20the%20best%20mosquito%20repellent%3F\" target=_blank>What is the best mosquito repellent?</a></h2><h3><a href=\"https://outdoors.stackexchange.com/questions/1116\" target=_blank> 1116 </a><strong>DEET</strong><em> (0.598)</em></h3><h3><a href=\"https://outdoors.stackexchange.com/questions/1056\" target=_blank> 1056 </a><strong>thiamine</strong><em> (0.335)</em></h3><h3><a href=\"https://outdoors.stackexchange.com/questions/569\" target=_blank> 569 </a><strong>Free-standing bug nets</strong><em> (0.154)</em></h3><h3><a href=\"https://outdoors.stackexchange.com/questions/1076\" target=_blank> 1076 </a><strong>Insect repellent is not 100% effective</strong><em> (0.052)</em></h3><h3><a href=\"https://outdoors.stackexchange.com/questions/829\" target=_blank> 829 </a><strong>bear-spray</strong><em> (0.046)</em></h3>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h2><a href=\"https://outdoors.stackexchange.com/search?q=How%20many%20miles%20can%20a%20person%20hike%20day%3F\" target=_blank>How many miles can a person hike day?</a></h2><h3><a href=\"https://outdoors.stackexchange.com/questions/17651\" target=_blank> 17651 </a><strong>20-25</strong><em> (0.318)</em></h3><h3><a href=\"https://outdoors.stackexchange.com/questions/19609\" target=_blank> 19609 </a><strong>12 miles</strong><em> (0.149)</em></h3><h3><a href=\"https://outdoors.stackexchange.com/questions/19558\" target=_blank> 19558 </a><strong>13</strong><em> (0.067)</em></h3><h3><a href=\"https://outdoors.stackexchange.com/questions/13030\" target=_blank> 13030 </a><strong>25-35</strong><em> (0.055)</em></h3><h3><a href=\"https://outdoors.stackexchange.com/questions/4536\" target=_blank> 4536 </a><strong>13 miles</strong><em> (0.022)</em></h3>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h2><a href=\"https://outdoors.stackexchange.com/search?q=How%20much%20water%20does%20a%20person%20need%20per%20day%3F\" target=_blank>How much water does a person need per day?</a></h2><h3><a href=\"https://outdoors.stackexchange.com/questions/1629\" target=_blank> 1629 </a><strong>3 liters</strong><em> (0.456)</em></h3><h3><a href=\"https://outdoors.stackexchange.com/questions/193\" target=_blank> 193 </a><strong>MINIMUM a gallon</strong><em> (0.244)</em></h3><h3><a href=\"https://outdoors.stackexchange.com/questions/20634\" target=_blank> 20634 </a><strong>0.4 to 0.6 L/day</strong><em> (0.208)</em></h3><h3><a href=\"https://outdoors.stackexchange.com/questions/11679\" target=_blank> 11679 </a><strong>4 litres</strong><em> (0.085)</em></h3><h3><a href=\"https://outdoors.stackexchange.com/questions/11687\" target=_blank> 11687 </a><strong>carry water</strong><em> (0.037)</em></h3>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ask('What is the best mosquito repellent?')\n",
    "ask('How many miles can a person hike day?')\n",
    "ask('How much water does a person need per day?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h2><a href=\"https://outdoors.stackexchange.com/search?q=What%20material%20are%20good%20walking%20poles%20made%20of%3F\" target=_blank>What material are good walking poles made of?</a></h2><h3><a href=\"https://outdoors.stackexchange.com/questions/13773\" target=_blank> 13773 </a><strong>CARBON AND ALUMINIUM</strong><em> (0.392)</em></h3><h3><a href=\"https://outdoors.stackexchange.com/questions/14112\" target=_blank> 14112 </a><strong>carbon fibre</strong><em> (0.231)</em></h3><h3><a href=\"https://outdoors.stackexchange.com/questions/14528\" target=_blank> 14528 </a><strong>train with poles</strong><em> (0.055)</em></h3><h3><a href=\"https://outdoors.stackexchange.com/questions/4471\" target=_blank> 4471 </a><strong>Trekking poles are a great helper if you have problems with your knees</strong><em> (0.043)</em></h3><h3><a href=\"https://outdoors.stackexchange.com/questions/5315\" target=_blank> 5315 </a><strong>BlackDiamond</strong><em> (0.017)</em></h3>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ask('What material are good walking poles made of?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h2><a href=\"https://outdoors.stackexchange.com/search?q=How%20many%20people%20can%20fit%20inside%20a%202-person%20tent%3F\" target=_blank>How many people can fit inside a 2-person tent?</a></h2><h3><a href=\"https://outdoors.stackexchange.com/questions/24593\" target=_blank> 24593 </a><strong>1</strong><em> (0.818)</em></h3><h3><a href=\"https://outdoors.stackexchange.com/questions/10225\" target=_blank> 10225 </a><strong>1</strong><em> (0.391)</em></h3><h3><a href=\"https://outdoors.stackexchange.com/questions/4433\" target=_blank> 4433 </a><strong>1 person</strong><em> (0.143)</em></h3><h3><a href=\"https://outdoors.stackexchange.com/questions/20803\" target=_blank> 20803 </a><strong>2-3</strong><em> (0.119)</em></h3><h3><a href=\"https://outdoors.stackexchange.com/questions/8791\" target=_blank> 8791 </a><strong>two people</strong><em> (0.044)</em></h3>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ask('How many people can fit inside a 2-person tent?')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Up next: [Chapter 15 - Foundation Models and Emerging Search Paradigms](../ch15/1.llm-exploration.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "vscode": {
   "interpreter": {
    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
